First of all, we will mention a bit about the owner of this dataset, Goodreads . Goodreads is a social cataloging website that allows individuals to freely search its database of books, annotations, and reviews.
This project aims to have an insight about different numerical parameters of the book and their relationship. Moreoever, alongside with the informaion given, we try to do a bit of web-scrapping and get another attribute (in this case, ‘Year’) in order to take our analysis further. Moreover, this would be my first milestone into Data Analysis, using all the skills and knowledges I have acquired during my summer time working with R.
This project is made possible by user Soumik on Kaggle, along with the wonderful reference from the Python notebook named Goodreads: Analysis and Recommending Books of Shivam Ralli. I would like to send my sincere thanks to both of them.
All of the datasets and scripts used will be uploaded along with this notebook.
First, we will load all the necessary libraries for this notebook.
library(ggplot2)
library(dplyr)
Attaching package: ‘dplyr’
The following objects are masked from ‘package:stats’:
filter, lag
The following objects are masked from ‘package:base’:
intersect, setdiff, setequal, union
library(RColorBrewer)
library(viridis)
Loading required package: viridisLite
library(WVPlots)
library(naniar)
library(e1071)
library(plotrix)
library(ggcorrplot)
library(waffle)
library(extrafont)
Registering fonts with R
library(GGally)
Attaching package: ‘GGally’
The following object is masked from ‘package:dplyr’:
nasa
library(tidyverse)
[30m── [1mAttaching packages[22m ────────────────────────────────────────────────────────────────────────────────────────────────────────── tidyverse 1.2.1 ──[39m
[30m[32m✔[30m [34mtibble [30m 2.1.1 [32m✔[30m [34mpurrr [30m 0.3.2
[32m✔[30m [34mtidyr [30m 0.8.3 [32m✔[30m [34mstringr[30m 1.4.0
[32m✔[30m [34mreadr [30m 1.3.1 [32m✔[30m [34mforcats[30m 0.4.0[39m
[30m── [1mConflicts[22m ───────────────────────────────────────────────────────────────────────────────────────────────────────────── tidyverse_conflicts() ──
[31m✖[30m [34mdplyr[30m::[32mfilter()[30m masks [34mstats[30m::filter()
[31m✖[30m [34mdplyr[30m::[32mlag()[30m masks [34mstats[30m::lag()[39m
library(cluster)
library(factoextra)
Welcome! Related Books: `Practical Guide To Cluster Analysis in R` at https://goo.gl/13EFCZ
library(rvest)
Loading required package: xml2
Attaching package: ‘rvest’
The following object is masked from ‘package:purrr’:
pluck
The following object is masked from ‘package:readr’:
guess_encoding
We import and investigate the structure of the dataset.
goodreads <- read.csv('/Users/ngohoanganh/Desktop/Goodreads Kaggle project/books.csv', stringsAsFactors = FALSE)
str(goodreads)
'data.frame': 13724 obs. of 10 variables:
$ bookID : int 1 2 3 4 5 8 9 10 12 13 ...
$ title : chr "Harry Potter and the Half-Blood Prince (Harry Potter #6)" "Harry Potter and the Order of the Phoenix (Harry Potter #5)" "Harry Potter and the Sorcerer's Stone (Harry Potter #1)" "Harry Potter and the Chamber of Secrets (Harry Potter #2)" ...
$ authors : chr "J.K. Rowling-Mary GrandPré" "J.K. Rowling-Mary GrandPré" "J.K. Rowling-Mary GrandPré" "J.K. Rowling" ...
$ average_rating : chr "4.56" "4.49" "4.47" "4.41" ...
$ isbn : chr "0439785960" "0439358078" "0439554934" "0439554896" ...
$ isbn13 : chr "9780439785969" "9780439358071" "9780439554930" "9780439554893" ...
$ language_code : chr "eng" "eng" "eng" "eng" ...
$ X..num_pages : chr "652" "870" "320" "352" ...
$ ratings_count : int 1944099 1996446 5629932 6267 2149872 38872 18 27410 3602 240189 ...
$ text_reviews_count: int 26249 27613 70390 272 33964 154 1 820 258 3954 ...
head(goodreads)
print(paste('The original goodreads book dataset has',nrow(goodreads),'rows and',ncol(goodreads),'columns.'))
[1] "The original goodreads book dataset has 13724 rows and 10 columns."
First of all, we know that this dataset has 13724 observations with 10 variables. There is one variable that is supposed to be written as “# num_pages”. However, as R cannot read special characters in column names, for the sake of convenience, we rename this variable as “num_pages”.
We obverse some problems with this dataset. At columns 4 and 8 (corresponding to variables “average_rating” and “num_page”), we can see that the observations are of type “character”, while they should be of type “integer”. We fix the problem with the function as.numeric in the following code chunk.
However, while fixing the problem above, those two columns yield some NA values. This can be due to those rows having too many commas, forcing the values that should belong to one columns move to another column. We omit these outliers for a clean dataset by using the function na.omit.
Moreover, one of the famous authors that we know, J.K Rowling, has a fairly long name of “J.K Rowling-Mary GrandPré”. After changing the author’s name, we have the dataset available for further analysis.
goodreads[goodreads=='J.K. Rowling-Mary GrandPré'] <- 'J.K. Rowling'
head(goodreads)
for(i in c(4,8)) {
goodreads[,i] <- as.numeric(goodreads[,i])
}
NAs introduced by coercionNAs introduced by coercion
names(goodreads)[names(goodreads) == "X..num_pages"] <- "num_pages"
goodreads <- na.omit(goodreads)
str(goodreads)
'data.frame': 13714 obs. of 10 variables:
$ bookID : int 1 2 3 4 5 8 9 10 12 13 ...
$ title : chr "Harry Potter and the Half-Blood Prince (Harry Potter #6)" "Harry Potter and the Order of the Phoenix (Harry Potter #5)" "Harry Potter and the Sorcerer's Stone (Harry Potter #1)" "Harry Potter and the Chamber of Secrets (Harry Potter #2)" ...
$ authors : chr "J.K. Rowling" "J.K. Rowling" "J.K. Rowling" "J.K. Rowling" ...
$ average_rating : num 4.56 4.49 4.47 4.41 4.55 4.78 3.69 4.73 4.38 4.38 ...
$ isbn : chr "0439785960" "0439358078" "0439554934" "0439554896" ...
$ isbn13 : chr "9780439785969" "9780439358071" "9780439554930" "9780439554893" ...
$ language_code : chr "eng" "eng" "eng" "eng" ...
$ num_pages : num 652 870 320 352 435 ...
$ ratings_count : int 1944099 1996446 5629932 6267 2149872 38872 18 27410 3602 240189 ...
$ text_reviews_count: int 26249 27613 70390 272 33964 154 1 820 258 3954 ...
- attr(*, "na.action")= 'omit' Named int 4011 4012 5688 5689 7057 7058 10603 10604 10671 10672
..- attr(*, "names")= chr "4011" "4012" "5688" "5689" ...
print(paste('The final goodreads book dataset has',nrow(goodreads),'rows and',ncol(goodreads),'columns.'))
[1] "The final goodreads book dataset has 13714 rows and 10 columns."
After fixing all the errors appeared in the dataset, we start to define the variables’ names. Their description is as follow:
We investigate whether there is any data cell missing with the vis_miss plot. Since we do not plot any significant in the plot, and it shows that less than 0.1% of the data is missing, we use gg_miss_var to get the exact number of data cells missing in each column. The result is, surprisingly, there is no data misising.
vis_miss(goodreads)
gg_miss_var(goodreads)
In this part, we will answer some of the interesting exploratory questions by visualizing data.
First, we will see the correlation between numerical columns of this dataset by using Pearson’s R, Kendall rank and Spearman correlation coefficient.
goodreads_corr <- subset(goodreads, select = c('average_rating','num_pages','ratings_count','text_reviews_count'))
goodreads_corr_pearson_round2 <- round(cor(goodreads_corr, method = "pearson", use = "pairwise.complete.obs"), 2)
data.frame(goodreads_corr_pearson_round2)
goodreads_corr_kendall_round2 <- round(cor(goodreads_corr, method = "kendall", use = "pairwise.complete.obs"), 2)
data.frame(goodreads_corr_kendall_round2)
goodreads_corr_spearman_round2 <- round(cor(goodreads_corr, method = "spearman", use = "pairwise.complete.obs"), 2)
data.frame(goodreads_corr_spearman_round2)
ggcorrplot(goodreads_corr_pearson_round2, hc.order = TRUE,
lab = TRUE,
title = "Pearson's R correlation matrix for Goodreads numerical variables",
outline.col = "white",
ggtheme = ggplot2::theme_gray,
colors = c("#6D9EC1", "white", "#E46726"))
ggcorrplot(goodreads_corr_kendall_round2, hc.order = TRUE,
lab = TRUE,
title = "Kendall correlation matrix for Goodreads numerical variables",
outline.col = "white",
ggtheme = ggplot2::theme_gray,
colors = c("#6D9EC1", "white", "#E46726"))
ggcorrplot(goodreads_corr_spearman_round2, hc.order = TRUE,
lab = TRUE,
title = "Spearman correlation matrix for Goodreads numerical variables",
outline.col = "white",
ggtheme = ggplot2::theme_gray,
colors = c("#6D9EC1", "white", "#E46726"))
From the three correlation matrices above, we can see that there is only one pair of numerical variables that are strongly correlated to each other, which is ratings_count and text_reviews_count. All of the correlation coefficients of these two variables, regardless of the method used, are always greater than 0.8, which can be considered to be a strong correlation. The remaining pairs of variables show little positive correlation, with all the remaining coefficients of less than 0.2, some are even less than 0.05.
This shows an intuitive result that the more readers rate a book on a scale of 0 to 5, the more readers will leave a written text review for the same book.
First, we will plot the histograms of the numerical/integer variables, including average_rating, num_pages, ratings_count and text_reviews_count.
ggplot(data = goodreads, aes(x = goodreads$average_rating)) +
geom_histogram(aes(y = ..density..),binwidth = 0.01, color='orange', fill='orange') +
geom_density(colour="blue", lwd = 0.5, alpha=0.5) +
labs(title="Histogram for books' average ratings", x = "Average ratings", y = "Number of books") +
geom_vline(data = goodreads, xintercept = mean(goodreads$average_rating, na.rm = TRUE), color = "red", linetype = "dashed", size = 0.5)
ggplot(data = goodreads, aes(x = goodreads$num_pages)) +
geom_histogram(aes(y = ..density..),binwidth = 10, color='orange', fill='orange') +
geom_density(colour="blue", lwd = 0.5, alpha=0.5) +
labs(title="Histogram for books' number of pages", x = "Number of pages", y = "Number of books") +
geom_vline(data = goodreads, xintercept = mean(goodreads$average_rating, na.rm = TRUE), color = "red", linetype = "dashed", size = 0.5)
ggplot(data = goodreads, aes(x = goodreads$ratings_count)) +
geom_histogram(aes(y = ..density..),binwidth = 100, color='orange', fill='orange') +
geom_density(colour="blue", lwd = 0.5, alpha=0.5) +
labs(title="Histogram for ratings count of each book/series", x = "Ratings count", y = "Number of books") +
geom_vline(data = goodreads, xintercept = mean(goodreads$average_rating, na.rm = TRUE), color = "red", linetype = "dashed", size = 0.5)
ggplot(data = goodreads, aes(x = goodreads$text_reviews_count)) +
geom_histogram(aes(y = ..density..),binwidth = 50 , color='orange', fill='orange') +
geom_density(colour="blue", lwd = 0.5, alpha=0.5) +
labs(title="Histogram for text reviews count for each book/series", x = "Text reviews count", y = "Number of books") +
geom_vline(data = goodreads, xintercept = mean(goodreads$average_rating, na.rm = TRUE), color = "red", linetype = "dashed", size = 0.5)
The histogram of average_ratings shows that most of the ratings lie between 3 and 5. The distribution of ratings between 3 and 5 seems to be familiar to a normal distribution (i.e most of the ratings focus around 4, and the density decreases upon reaching the edges). Later, we will investigate this distribution and test how close is it to the normal distribution.
With the histogram of num_pages, most of the books have less than 1000 pages, with many of them having 250-500 pages. Later, we will also zoom into this part of the histogram to have a closer look of the distribution.
The histograms of ratings_count and text_reviews_count is so much heavily skewed to the left that we can only see a straight small bar at the position 0, and the density curve decreases to 0 shortly. As such, to further investigate the distribution of these variables, we have to later take a closer look at the positions around 0.
First, as R does not have a built in function to calculate the mode of a dataset, we proceed to define that function ourselves as follow.
# Define the function getmode in R to get the mode of data (appearance with highest frequency)
getmode <- function(v) {
uniqv <- unique(v)
uniqv[which.max(tabulate(match(v, uniqv)))]
}
Now, we will generate a table of basic properties of each variable.
basic_statistics_goodreads <- data.frame(matrix(NA, ncol = 13, nrow = 10))
names(basic_statistics_goodreads) <- c('Variable',
'Minimum score',
'25th percentile',
'Median',
'Mean score',
'75th percentile',
'Maximum score',
'Value with highest frequency (mode)',
'Frequency of mode',
'Standard Deviation',
'SE Mean',
'Skewness',
'Kurtosis')
for(i in c(4,8:10)) {
basic_statistics_goodreads[i,1] <- colnames(goodreads)[i]
basic_statistics_goodreads[i,2] <- min(goodreads[,i], na.rm = TRUE)
basic_statistics_goodreads[i,3] <- quantile(goodreads[,i], 0.25, na.rm = TRUE)
basic_statistics_goodreads[i,4] <- quantile(goodreads[,i], 0.5, na.rm = TRUE)
basic_statistics_goodreads[i,5] <- mean(goodreads[,i], na.rm = TRUE)
basic_statistics_goodreads[i,6] <- quantile(goodreads[,i], 0.75, na.rm = TRUE)
basic_statistics_goodreads[i,7] <- max(goodreads[,i], na.rm = TRUE)
basic_statistics_goodreads[i,8] <- getmode(na.omit(goodreads[,i]))
basic_statistics_goodreads[i,9] <- sum(goodreads[,i] == basic_statistics_goodreads[i,8])
basic_statistics_goodreads[i,10] <- sd(goodreads[,i], na.rm = TRUE)
basic_statistics_goodreads[i,11] <- std.error(goodreads[,i], na.rm = TRUE)
basic_statistics_goodreads[i,12] <- skewness(goodreads[,i], na.rm = TRUE)
basic_statistics_goodreads[i,13] <- kurtosis(goodreads[,i], na.rm = TRUE)
}
basic_statistics_goodreads <- na.omit(basic_statistics_goodreads)
View(basic_statistics_goodreads)
First of all, we see an interesting fact that the mode of text_reviews_count is 0 (more than 900 books receive 0 text reviews), which means that books having 0 text reviews occur the most frequently in the dataset. Moreover, the mode of average_ratings is also exactly 4.0, which shows that readers are really generous with the books they are reading.
Among the 4 variables, there is only one variable (average_ratings) that has a negative skewness. This is due to the fact that the histogram of this variable skews heavily to the right, with most of the observations lie between 3 and 5. The three remaining variables has a significantly positive skewness, which means that these variables skew heavily to the left. This can be verified by looking at the histograms of these variables.
We also note that the 75th percentile of each variable, apart from average_ratings, is relatively very small comparing to the maximum, which can also explain the positive skewness of those variables.
All 4 variables have a positive kurtosis, which shows that the distribution of these variables have heavier tails than the normal distribution with the same mean and standard deviation.
From the properties witnessed above, to better understand the distribution of variables within specific ranges, we again generate histograms for two variables, ratings_count and text_reviews_count. This time, we will set the limit of the x-axis to be 100 for num_pages, 5000 for ratings_count and 500 for text_reviews_count.
ggplot(data = goodreads, aes(x = goodreads$num_pages)) +
geom_histogram(aes(y = ..density..),binwidth = 20, color='orange', fill='orange') +
xlim(c(0,1000)) +
geom_density(colour="blue", lwd = 0.5, alpha=0.5) +
labs(title="Histogram for books' number of pages", x = "Number of pages", y = "Number of books") +
geom_vline(data = goodreads, xintercept = mean(goodreads$average_rating, na.rm = TRUE), color = "red", linetype = "dashed", size = 0.5)
ggplot(data = goodreads, aes(x = goodreads$ratings_count)) +
geom_histogram(aes(y = ..density..),binwidth = 10, color='orange', fill='orange') +
xlim(c(0,5000)) +
geom_density(colour="blue", lwd = 0.5, alpha=0.5) +
labs(title="Histogram for ratings count of each book/series", x = "Ratings count", y = "Number of books") +
geom_vline(data = goodreads, xintercept = mean(goodreads$average_rating, na.rm = TRUE), color = "red", linetype = "dashed", size = 0.5)
ggplot(data = goodreads, aes(x = goodreads$text_reviews_count)) +
geom_histogram(aes(y = ..density..),binwidth = 1 , color='orange', fill='orange') +
xlim(c(0,500)) +
geom_density(colour="blue", lwd = 0.5, alpha=0.5) +
labs(title="Histogram for text reviews count for each book/series", x = "Text reviews count", y = "Number of books") +
geom_vline(data = goodreads, xintercept = mean(goodreads$average_rating, na.rm = TRUE), color = "red", linetype = "dashed", size = 0.5)
After plotting the histograms of three variables again, we see that the ratings_count and text_reviews_count still skew heavily to the left. The mode of these two variables 0 and 3, respectively, which also add to the fact that most books receive very few ratings/reviews.
Considering number of pages, most books have the number of pages of around 175 - 325. Then, the number of book inversely proportional to the number of pages (i.e the number of books is decreasing with respect to their length).
ggpairs(with(goodreads, data.frame(average_rating, num_pages, ratings_count, text_reviews_count)))
From this scatterplot matrix, we can see that, apart from the pairs ratings_count and text_reviews_count, the remaining shows little to no correlation. All of the scatterplot shows a nearly vertical trend line (i.e there is no linear correlation between the two variables plotted).
First, we divide the ratings into five different ranges:
We calculate the number of books within each range, along with their percentages.
rating_distribution_goodreads <- data.frame(matrix(0, ncol = 3, nrow = 5))
names(rating_distribution_goodreads) <- c('Range',
'number_of_books',
'percentage')
for(i in c(1:5)) {
rating_distribution_goodreads[i,1] <- paste('Between',i-1,'and',i)
}
for(i in c(1:nrow(goodreads))) {
if (goodreads[i,4] >= 0 & goodreads[i,4] < 1) {
rating_distribution_goodreads[1,2] = rating_distribution_goodreads[1,2] + 1
}
if (goodreads[i,4] >= 1 & goodreads[i,4] < 2) {
rating_distribution_goodreads[2,2] = rating_distribution_goodreads[2,2] + 1
}
if (goodreads[i,4] >= 2 & goodreads[i,4] < 3) {
rating_distribution_goodreads[3,2] = rating_distribution_goodreads[3,2] + 1
}
if (goodreads[i,4] >= 3 & goodreads[i,4] < 4) {
rating_distribution_goodreads[4,2] = rating_distribution_goodreads[4,2] + 1
}
if (goodreads[i,4] >= 4 & goodreads[i,4] < 5) {
rating_distribution_goodreads[5,2] = rating_distribution_goodreads[5,2] + 1
}
}
rating_distribution_goodreads$percentage <- round((rating_distribution_goodreads$number_of_books)/nrow(goodreads)*100,digits = 2)
rating_distribution_goodreads
Then, we will plot two different pie charts, a traditional one and a square pie chart, to observe the distribution of books within ranges.
ggplot(data=rating_distribution_goodreads, aes(x="", y=percentage,
fill = factor(rating_distribution_goodreads[,1]), )) +
geom_bar(width = 1, stat = "identity") +
ggtitle("Pie chart of percentage of Books with respect to ratings count") +
coord_polar(theta="y", start = 0) +
labs(fill = factor(rating_distribution_goodreads[,1]))
rating_distribution_number <- c(`Between 0 and 1`= 0.25, `Between 1 and 2`= 0.02, `Between 2 and 3`= 0.53, `Between 3 and 4`= 55.32, `Between 4 and 5`= 43.69)
waffle(rating_distribution_number, rows=6,
colors=brewer.pal(5,"Set1"),
title="Percentage of Books with respect to ratings count")
We can see that, more than half of the books are rated between 3 and 4 points, while most the remaining books are ranked between 4 and 5. There are a total of 13577 books rated between 3 and 5, which counts up to 99% of the books in the dataset.
book_frequency <- data.frame(table(goodreads$title), stringsAsFactors = FALSE)
names(book_frequency) <- c('title','frequency')
book_frequency <- book_frequency[order(-book_frequency$frequency),]
head(book_frequency)
book_frequency_top20 <- book_frequency[c(1:20),]
ggplot(data=book_frequency_top20, aes(x=reorder(title, frequency),y=frequency, fill = factor(c(1:nrow(book_frequency_top20))))) +
theme(legend.position = "none") +
geom_bar(stat="identity") +
coord_flip() +
ggtitle('20 books with highest frequency') +
labs(x = "Title", y = "Frequency")
Above, we have plotted the 20 books with the most appearances in the dataset. We can see that “One Hundred Year of Solitude” and “Salem’s Lot” are the two books appearing the most, with 11 times for each book.
These books are published again and again, each time with another publisher. This shows that these are still in great demand, despite the flow of time.
book_language <- data.frame(table(goodreads$language_code), stringsAsFactors = FALSE)
names(book_language) <- c('language','frequency')
book_language <- book_language[order(-book_language$frequency),]
book_language$percentage <- round(book_language$frequency/nrow(goodreads)*100, digits = 2)
head(book_language)
ggplot(book_language, aes(x=reorder(language,-frequency),y=frequency, fill = factor(c(1:nrow(book_language))))) +
theme(legend.position = "none") +
geom_bar(stat="identity") +
ggtitle('Languages written in books/series') +
labs(x = "Language", y = "Total number of books") +
geom_text(aes(label=frequency), vjust=-0.5)
We can easily see that most of the books in this dataset is written in English. Two most used languages are English and English-US, while within 6 most commonly used languages, English and its different versions (eng-US and eng-GB) counts up to half of them. Combining all the different versions of this language, English is written in 12634 books, which is 92,12% of all the books. Apart from English, Spanish, the third commonly used language, only has 419 books, which is 3.1% of the books in this dataset.
book_average_rating <- goodreads[order(-goodreads$average_rating),]
book_average_rating_top10 <- book_average_rating[c(1:10),]
book_average_rating_top10
ggplot(data=book_average_rating_top10, aes(x=reorder(title, average_rating),y=average_rating, fill = factor(c(1:nrow(book_average_rating_top10))))) +
theme(legend.position = "none") +
geom_bar(stat="identity") +
coord_flip() +
scale_fill_brewer(palette="Paired") +
ggtitle('Books/series with highest average rating') +
labs(x = "Book/series", y = "Average rating") +
geom_text(
aes(label = average_rating), colour = "black",
hjust = -0.1, size = 2,
position = position_dodge(width = 1),
inherit.aes = TRUE
)
author_highest_book_average_rating <- data.frame(table(book_average_rating_top10$authors))
names(author_highest_book_average_rating) <- c('author', 'number_of_books')
author_highest_book_average_rating <- author_highest_book_average_rating[order(-author_highest_book_average_rating$number_of_books),]
author_highest_book_average_rating
Surprisingly, all 10 of the highest rating books have the maximum rating, 5. As a result, we will investigate all the books that have the rating of 5 in this dataset.
books_rating_5 <- goodreads[goodreads$average_rating == 5,]
books_rating_5 <- books_rating_5[order(-books_rating_5$ratings_count),]
books_rating_5
print(paste('There are',nrow(books_rating_5),'books that receive an average rating of 5.'))
[1] "There are 28 books that receive an average rating of 5."
We can easily see that all of the books with average ratings of 5 receive very few ratings. The maximum ratings count among these books are 5. Specifically, there are 4 books that receive 0 ratings, and still receive an average rating of 5, which is not supposed to happen. Moreover, 23/28 books in the list receive 0 text reviews. We can not consider these books to be the ones with highest average rating, as the ratings count is too low.
Instead, we propose the following method: We only consider the average rating of books with more than 1000 ratings.
book_1000_ratings_count <- goodreads[goodreads$ratings_count >= 1000,]
book_1000_ratings_count <- book_1000_ratings_count[order(-book_1000_ratings_count$average_rating),]
book_1000_ratings_count
book_1000_ratings_count_top10 <- book_1000_ratings_count[c(1:10),]
book_1000_ratings_count_top10
ggplot(data=book_1000_ratings_count_top10, aes(x=reorder(title, average_rating),y=average_rating, fill = factor(c(1:nrow(book_1000_ratings_count_top10))))) +
theme(legend.position = "none") +
geom_bar(stat="identity") +
coord_flip() +
scale_fill_brewer(palette="Paired") +
ggtitle('Books/series with highest average rating (ratings count >= 1000)') +
labs(x = "Book/series", y = "Average rating") +
geom_text(
aes(label = average_rating), colour = "black",
hjust = -0.1, size = 2,
position = position_dodge(width = 1),
inherit.aes = TRUE
)
author_book_1000_ratings_count <- data.frame(table(book_1000_ratings_count_top10$authors))
names(author_book_1000_ratings_count) <- c('author', 'number_of_books')
author_book_1000_ratings_count <- author_book_1000_ratings_count[order(-author_book_1000_ratings_count$number_of_books),]
author_book_1000_ratings_count
We can see that, there is only 6059 books that have the ratings count of more than or equal to 1000, which accounts for 44.18% of all the books in the dataset.
Within these 6059 books, the highest has the average rating of 4.82, while the lowest rating among the top 10 books is 4.7.
Among those 10 books, half of them are written by Bill Watterson, two are from J.K. Rowling, one from Patrick O’Brian, one from Joyce Meyer and one from anonymous. We can easily see that Bill Watterson has written books that receive very high ratings from a large number of readers. Two among the top 10 from J.K Rowling are both sets of Harry Potter (one set of #1-5 and one set of #1-6).
Another interesting fact is that all books in top 10 are written in English, with one specifically written in en-US.
book_num_pages <- goodreads[order(-goodreads$num_pages),]
book_num_pages_top10 <- book_num_pages[c(1:10),]
book_num_pages_top10
ggplot(data=book_num_pages_top10, aes(x=reorder(title, num_pages),y=num_pages, fill = factor(c(1:nrow(book_num_pages_top10))))) +
theme(legend.position = "none") +
geom_bar(stat="identity") +
coord_flip() +
scale_fill_brewer(palette="Paired") +
ggtitle('Books/series with highest number of pages') +
labs(x = "Book/series", y = "Number of pages") +
geom_text(
aes(label = num_pages), colour = "black",
hjust = -0.1, size = 2,
position = position_dodge(width = 1),
inherit.aes = TRUE
)
author_book_num_pages <- data.frame(table(book_num_pages_top10$authors))
names(author_book_num_pages) <- c('author', 'number_of_books')
author_book_num_pages <- author_book_num_pages[order(-author_book_num_pages$number_of_books),]
author_book_num_pages
We can easily see that, all of the 10 longest in this dataset are all series. Among them, “The Complet Aubrey/Maturin Novels” stands out from the crowd, with its length of nearly 3 times the length of the 10th longest series, “Civil War: a Narrative”. Moreover, “The Norton Anthology of English Literature” has 3 of their volumns presenting in the top 10 (Vols A-C, Vol 2 and Vol 1). However, their appearances are not consistent: Vols A-C and Vol 1 have the same name, but their number of pages are different.
book_ratings_count <- goodreads[order(-goodreads$ratings_count),]
book_ratings_count_top10 <- book_ratings_count[c(1:10),]
book_ratings_count_top10
ggplot(data=book_ratings_count_top10, aes(x=reorder(title, ratings_count),y=ratings_count, fill = factor(c(1:nrow(book_ratings_count_top10))))) +
theme(legend.position = "none") +
geom_bar(stat="identity") +
coord_flip() +
scale_fill_brewer(palette="Paired") +
ggtitle('Books/series with highest number of ratings count') +
labs(x = "Book/series", y = "Number of ratings") +
geom_text(
aes(label = ratings_count), colour = "black",
hjust = -0.1, size = 2,
position = position_dodge(width = 1),
inherit.aes = TRUE
)
author_most_book_ratings_count <- data.frame(table(book_ratings_count_top10$authors))
names(author_most_book_ratings_count) <- c('author', 'number_of_books')
author_most_book_ratings_count <- author_most_book_ratings_count[order(-author_most_book_ratings_count$number_of_books),]
author_most_book_ratings_count
There is a huge gap between the first two most rated books and the remainings. Specifically speaking, there is a 2 million rating gap between Twilight #1 - the second most rated book and “The Hobbit or Three and Back Again” - the third most rated book. Even between the first position, “Harry Potter and the Sorcerer’s Stone” and the second one, there is also a significant gap of about 1.3 million ratings. However, between the third and the 10th position, the gap is just about 350.000 ratings.
About the authors’ list of top 10 books, we can easily see that J.K Rowling and her Harry Potter series are dominating the ratings count, with 4 of her books in the series making their ways to the top 10. - Harry Potter and the Sorcerer’s Stone (Harry Potter #1) at first place - Harry Potter and the Prisoner of Azkaban (Harry Potter #3) at sixth place - Harry Potter and the Chamber of Secrets (Harry Potter #2) at seventh place - Harry Potter and the Order of the Phoenix (Harry Potter #5) at tenth place
Large number of ratings count, alongside with high average rating (all over 4.4), these definitely shows how successful this series have been.
book_text_reviews_count <- goodreads[order(-goodreads$text_reviews_count),]
book_text_reviews_count_top10 <- book_text_reviews_count[c(1:10),]
book_text_reviews_count_top10
ggplot(data=book_text_reviews_count_top10, aes(x=reorder(title, text_reviews_count),y=text_reviews_count, fill = factor(c(1:nrow(book_text_reviews_count_top10))))) +
theme(legend.position = "none") +
geom_bar(stat="identity") +
coord_flip() +
scale_fill_brewer(palette="Paired") +
ggtitle('Books/series with highest number of text reviews') +
labs(x = "Book/series", y = "Number of text reviews") +
geom_text(
aes(label = text_reviews_count), colour = "black",
hjust = -0.1, size = 2,
position = position_dodge(width = 1),
inherit.aes = TRUE
)
author_most_book_text_reviews_count <- data.frame(table(book_text_reviews_count_top10$authors))
names(author_most_book_text_reviews_count) <- c('author', 'number_of_books')
author_most_book_text_reviews_count <- author_most_book_text_reviews_count[order(-author_most_book_text_reviews_count$number_of_books),]
author_most_book_text_reviews_count
Once again, there is a significant gap between the first four positions; after that, the gap is getting much narrower.
Moreover, this time, the author’s list of these books are more widely spread, with 10 different authors, each having one of their books in the top 10 list.
We have seen some familiar names that appear in both top 10 of ratings count and text reviews count. The question is, how many of them are there, and what is the difference between their performance in these two criteria?
common_top10 <- intersect(book_text_reviews_count_top10, book_ratings_count_top10)
common_top10
Although there is a very strong positive correlation between the two variables, there are only two books that belong to both top 10 of these two criteria. They are:
These two books are performing extremely well in both criteria, and so far the leading books in this dataset, considering the number of ratings/reviews:
author_most_books <- data.frame(table(goodreads$authors), stringsAsFactors = FALSE)
names(author_most_books) <- c('author','total_number_of_books')
author_most_books <- author_most_books[order(-author_most_books$total_number_of_books),]
head(author_most_books)
author_most_books_top10 <- author_most_books[c(1:10),]
ggplot(data=author_most_books_top10, aes(x=reorder(author, total_number_of_books),y=total_number_of_books, fill = factor(c(1:nrow(author_most_books_top10))))) +
theme(legend.position = "none") +
geom_bar(stat="identity") +
coord_flip() +
scale_fill_brewer(palette="Set3") +
ggtitle('Author with greatest number of books appeared') +
labs(x = "Author", y = "Total number of books") +
geom_text(
aes(label = total_number_of_books), colour = "black",
hjust = -0.5, size = 3,
position = position_dodge(width = 1),
inherit.aes = TRUE
)
author_most_books_top10
Once again, the first two authors creates a huge gap with the remaining. Both Agatha Christie and Stephen King have more than 65 books, which is at least 18 books more than the third author, Orson Scott Card.
K-means clustering is the most commonly used unsupervised machine learning algorithm for partitioning a given data set into a set of k groups (i.e. k clusters), where k represents the number of groups pre-specified by the analyst.
First, we will build a model to cluster this dataset based on two variables: average_rating and ratings_count.
To determine the optimal number o clusters, we use the two following popular methods:
goodreads_kmeans1 <- goodreads[,c('average_rating','ratings_count')]
set.seed(123)
fviz_nbclust(goodreads_kmeans1, kmeans, method = "wss")
fviz_nbclust(goodreads_kmeans1, kmeans, method = "silhouette")
From both of the two approaches above, the optimal number of clusters seem to be at 5. As a result, we will perform an analysis and extract the results with 5 clusters.
set.seed(123)
final_kmeans1 <- kmeans(goodreads_kmeans1, 5, nstart = 25)
print(final_kmeans1)
K-means clustering with 5 clusters of sizes 23, 13257, 357, 75, 2
Cluster means:
average_rating ratings_count
1 4.100000 1760033.261
2 3.927476 5645.072
3 4.015490 191223.644
4 4.027733 667374.147
5 4.030000 4998636.500
Clustering vector:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29
1 1 5 2 1 2 2 2 2 3 2 2 2 3 2 2 2 2 2 2 2 2 2 2 2 1 2 2 2
30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58
2 2 2 3 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87
2 2 2 2 2 2 3 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116
2 2 2 2 2 2 2 2 3 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174
3 2 2 2 2 2 2 2 4 2 2 2 2 2 2 2 3 2 2 2 3 2 2 2 2 2 2 2 2
175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203
2 2 3 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 2 2 2 2 2 3 2 2 3 2 2
262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290
2 3 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319
2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 2 2 2 2 2 2 2 2 1 2 2 2 3 2
320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348
2 2 2 2 2 1 2 2 3 2 3 1 2 2 2 2 2 2 2 2 2 1 2 2 1 2 2 2 4
349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 2 2 2 3 2 2 2 3 2 3 2 2 2
378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 4 2 2 2 2 2 2 2 2 3 2 2 3
407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435
2 2 2 2 2 2 2 3 2 2 2 2 2 3 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 2 2 2 2 2 2
465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493
4 2 2 2 2 2 2 2 2 2 2 2 2 4 2 2 2 2 2 2 2 2 3 3 2 2 2 2 2
494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 2 2
552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 4 4 2 3 2 3 2 2
581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 2 4 2 2
639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667
2 2 3 2 2 3 2 2 2 2 2 2 2 2 2 2 2 2 2 3 2 2 2 2 2 2 1 2 2
668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696
2 2 2 2 4 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754
2 2 2 2 2 2 2 2 2 2 2 2 2 3 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3
755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783
2 4 2 2 2 2 3 2 2 2 2 2 2 2 4 3 2 3 2 2 2 2 2 2 2 2 2 2 2
784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841
2 2 2 2 2 2 2 2 2 2 2 3 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870
2 2 2 2 2 2 2 3 2 2 2 2 2 2 2 2 2 2 2 3 2 2 2 2 2 2 2 2 2
871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899
2 2 2 2 3 2 2 2 2 2 2 2 2 2 2 2 2 4 2 2 2 4 2 2 2 2 2 2 2
900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928
2 2 2 2 2 2 2 2 2 2 2 3 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957
2 3 2 2 2 2 2 2 2 3 2 2 2 2 3 2 2 2 2 2 2 2 2 2 2 2 2 2 2
958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986
2 2 2 2 3 2 2 2 2 2 2 2 2 2 4 2 2 2 2 2 2 2 2 2 3 2 2 2 2
987 988 989 990 991 992 993 994 995 996 997 998 999 1000
2 2 2 2 2 2 2 2 2 2 2 2 2 2
[ reached getOption("max.print") -- omitted 12714 entries ]
Within cluster sum of squares by cluster:
[1] 2.834358e+12 2.385364e+12 2.431455e+12 2.747742e+12 7.970680e+11
(between_SS / total_SS = 93.6 %)
Available components:
[1] "cluster" "centers" "totss" "withinss" "tot.withinss" "betweenss" "size" "iter" "ifault"
str(final_kmeans1)
List of 9
$ cluster : Named int [1:13714] 1 1 5 2 1 2 2 2 2 3 ...
..- attr(*, "names")= chr [1:13714] "1" "2" "3" "4" ...
$ centers : num [1:5, 1:2] 4.1 3.93 4.02 4.03 4.03 ...
..- attr(*, "dimnames")=List of 2
.. ..$ : chr [1:5] "1" "2" "3" "4" ...
.. ..$ : chr [1:2] "average_rating" "ratings_count"
$ totss : num 1.75e+14
$ withinss : num [1:5] 2.83e+12 2.39e+12 2.43e+12 2.75e+12 7.97e+11
$ tot.withinss: num 1.12e+13
$ betweenss : num 1.64e+14
$ size : int [1:5] 23 13257 357 75 2
$ iter : int 6
$ ifault : int 0
- attr(*, "class")= chr "kmeans"
fviz_cluster(final_kmeans1, data = goodreads_kmeans1)
We can see that the data are clearly classified into 5 different clusters. These clusters can be ordered by ratings count (i.e, the mean ratings count of these clusters can be ordered in a increasing order with clear differences).
As the rating count decreases, the average rating are more widely spread (i.e the average rating rate is higher), which means that the average rating is neither concentrated nor correct.
We also can extract the clusters and add to our initial data for futher analysis at the cluster level:
goodreads_kmeans1_result <- goodreads %>%
mutate(Cluster = final_kmeans1$cluster)
goodreads_kmeans1_result
Now, we will try to apply K-means algorithm for all of the numerical variables of this dataset.
goodreads_kmeans2 <- goodreads[,c('average_rating','num_pages','ratings_count','text_reviews_count')]
set.seed(123)
fviz_nbclust(goodreads_kmeans2, kmeans, method = "wss")
fviz_nbclust(goodreads_kmeans2, kmeans, method = "silhouette")
After considering the elbow plot and sihouette plot, we once again come up with the optimal number of clusters of 5. As a result, we will again cluster our dataset into 5 groups.
set.seed(123)
final_kmeans2 <- kmeans(goodreads_kmeans2, 5, nstart = 25)
print(final_kmeans2)
K-means clustering with 5 clusters of sizes 2, 75, 13257, 23, 357
Cluster means:
average_rating num_pages ratings_count text_reviews_count
1 4.030000 409.0000 4998636.500 82004.5000
2 4.027733 395.6133 667374.147 15690.9467
3 3.927476 340.7500 5645.072 239.9849
4 4.100000 409.2174 1760033.261 32802.3478
5 4.015490 387.9188 191223.644 5718.3950
Clustering vector:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29
4 4 1 3 4 3 3 3 3 5 3 3 3 5 3 3 3 3 3 3 3 3 3 3 3 4 3 3 3
30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58
3 3 3 5 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87
3 3 3 3 3 3 5 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116
3 3 3 3 3 3 3 3 5 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174
5 3 3 3 3 3 3 3 2 3 3 3 3 3 3 3 5 3 3 3 5 3 3 3 3 3 3 3 3
175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203
3 3 5 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 5 3 3 3 3 3 5 3 3 5 3 3
262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290
3 5 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319
3 3 3 3 3 3 3 3 3 3 3 3 3 3 5 3 3 3 3 3 3 3 3 4 3 3 3 5 3
320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348
3 3 3 3 3 4 3 3 5 3 5 4 3 3 3 3 3 3 3 3 3 4 3 3 4 3 3 3 2
349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 5 3 3 3 5 3 3 3 5 3 5 3 3 3
378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 2 3 3 3 3 3 3 3 3 5 3 3 5
407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435
3 3 3 3 3 3 3 5 3 3 3 3 3 5 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 5 3 3 3 3 3 3
465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493
2 3 3 3 3 3 3 3 3 3 3 3 3 2 3 3 3 3 3 3 3 3 5 5 3 3 3 3 3
494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 5 3 3
552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 2 2 3 5 3 5 3 3
581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 5 3 2 3 3
639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667
3 3 5 3 3 5 3 3 3 3 3 3 3 3 3 3 3 3 3 5 3 3 3 3 3 3 4 3 3
668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696
3 3 3 3 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754
3 3 3 3 3 3 3 3 3 3 3 3 3 5 3 3 3 3 3 3 3 3 3 3 3 3 3 3 5
755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783
3 2 3 3 3 3 5 3 3 3 3 3 3 3 2 5 3 5 3 3 3 3 3 3 3 3 3 3 3
784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812
3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841
3 3 3 3 3 3 3 3 3 3 3 5 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870
3 3 3 3 3 3 3 5 3 3 3 3 3 3 3 3 3 3 3 5 3 3 3 3 3 3 3 3 3
871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899
3 3 3 3 5 3 3 3 3 3 3 3 3 3 3 3 3 2 3 3 3 2 3 3 3 3 3 3 3
900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928
3 3 3 3 3 3 3 3 3 3 3 5 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957
3 5 3 3 3 3 3 3 3 5 3 3 3 3 5 3 3 3 3 3 3 3 3 3 3 3 3 3 3
958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986
3 3 3 3 5 3 3 3 3 3 3 3 3 3 2 3 3 3 3 3 3 3 3 3 5 3 3 3 3
987 988 989 990 991 992 993 994 995 996 997 998 999 1000
3 3 3 3 3 3 3 3 3 3 3 3 3 3
[ reached getOption("max.print") -- omitted 12714 entries ]
Within cluster sum of squares by cluster:
[1] 7.973378e+11 2.754288e+12 2.390668e+12 2.840440e+12 2.436637e+12
(between_SS / total_SS = 93.6 %)
Available components:
[1] "cluster" "centers" "totss" "withinss" "tot.withinss" "betweenss" "size" "iter" "ifault"
fviz_cluster(final_kmeans2, data = goodreads_kmeans2)
Surprising enough, we receive the exactly same result as we do with only two variables, with every observation being in the same group as the first model.
As we have has ISBNs associated with books, we can retrieve the year when the book was written by web-scrapping from the website as follow.
goodreads_year <- goodreads
goodreads_year$Year <- 0
for (i in c(1:100)) {
url_i <- paste('https://isbndb.com/book/',goodreads[i,'isbn'],sep = "")
webpage_i <- read_html(url_i)
year_data_html <- html_nodes(webpage_i,'td')
year_data <- html_text(year_data_html)
goodreads_year[i,'Year'] <- as.numeric(year_data[6])
}
goodreads_year <- goodreads[goodreads$Year >= 1900,]
goodreads_year <- na.omit(goodreads_year)
goodreads_year
However, this code in R may take a little while to code. As a result, we will proceed with a script in Python, takean from the “Goodreads: Analysis and Recommending Books” notebook on Kaggle.
We can easily see some NAs appearing in the column “Year”. This is due to the fact that the website we are scrapping data from lacks the information “year”. In order to further analyze this dataset with the variable “Year”, we omit all of NAs.
However, instead of doing web-scrapping with R, we decided to use a much more effective tool, Python. The Python code for webscrapping can be found within the same file of this notebook. After running the Python script, we save the csv file as goodreads_year.csv, with encoding “UTF-16”. We import the file as follow.
goodreads_year <- read.csv('/Users/ngohoanganh/Desktop/Goodreads Kaggle project/goodreads_year.csv', stringsAsFactor = FALSE, fileEncoding = "UTF-16")
str(goodreads_year)
'data.frame': 13013 obs. of 11 variables:
$ bookID : int 1 2 3 4 5 8 9 10 12 13 ...
$ title : chr "Harry Potter and the Half-Blood Prince (Harry Potter #6)" "Harry Potter and the Order of the Phoenix (Harry Potter #5)" "Harry Potter and the Sorcerer's Stone (Harry Potter #1)" "Harry Potter and the Chamber of Secrets (Harry Potter #2)" ...
$ authors : chr "J.K. Rowling-Mary GrandPré" "J.K. Rowling-Mary GrandPré" "J.K. Rowling-Mary GrandPré" "J.K. Rowling" ...
$ average_rating : num 4.56 4.49 4.47 4.41 4.55 4.78 3.69 4.73 4.38 4.38 ...
$ isbn : chr "0439785960" "0439358078" "0439554934" "0439554896" ...
$ isbn13 : num 9.78e+12 9.78e+12 9.78e+12 9.78e+12 9.78e+12 ...
$ language_code : chr "eng" "eng" "eng" "eng" ...
$ X..num_pages : int 652 870 320 352 435 2690 152 3342 815 815 ...
$ ratings_count : int 1944099 1996446 5629932 6267 2149872 38872 18 27410 3602 240189 ...
$ text_reviews_count: int 26249 27613 70390 272 33964 154 1 820 258 3954 ...
$ Year : int 2006 2004 1997 2003 2004 2004 2005 2005 2005 2002 ...
print(paste('There are',nrow(goodreads_year),'books whose `Year` attribute can be retrieved, which accounts for',round(nrow(goodreads_year)/nrow(goodreads)*100, digits = 2),'eprcent of the original dataset.'))
[1] "There are 13013 books whose `Year` attribute can be retrieved, which accounts for 94.89 eprcent of the original dataset."
After retrieving the Year variable, we will proceed to analyse the new dataset with the following graphs:
mean_year <- aggregate(average_rating ~ Year, data = goodreads_year, FUN = mean)
mean_year
ggplot(data = mean_year, aes(x = Year, y = average_rating)) +
geom_line(color = "#FC4E07", size = 1) +
labs(x = "Year", y = "Average rating", title = "Average rating for each year") +
geom_smooth(method = "lm")
goodreads_year[goodreads_year$Year == 1922,]
goodreads_year[goodreads_year$Year == 1931,]
From the graph, we can see two opposite peaks appearing:
Apart from the peaks mentioned above, there is no other abnormal points within the graph. For the remaining years, the average rating fluctuate around 4.0.
total_books_year <- goodreads_year %>% count(Year)
names(total_books_year) <- c('Year','number_of_books')
total_books_year
ggplot(data = total_books_year, aes(x = Year, y = number_of_books)) +
geom_line(color = "#00AFBB", size = 1) +
labs(x = "Year", y = "Total number of books", title = "Number of books written for each year") +
geom_smooth(method = "lm")
We can easily see that, before 1975, there are not many books written in each year. However, after that timestamp, the books published every year start to increase gradually, and skyrocketted between 1990 and 2006. After reaching its peak, the number of books suddenly dropped significantly, and since 2016, there are less than 10 new books written each year.
sum_ratings_count_year <- aggregate(ratings_count ~ Year, data = goodreads_year, FUN = sum)
sum_ratings_count_year
ggplot(data = sum_ratings_count_year, aes(x = Year, y = ratings_count)) +
geom_line(color = "purple", size = 1) +
labs(x = "Year", y = "Total ratings count", title = "Total ratings count of all books for each year") +
geom_smooth(method = "lm")
average_ratings_count_year <- aggregate(ratings_count ~ Year, data = goodreads_year, FUN = mean)
average_ratings_count_year
ggplot(data = average_ratings_count_year, aes(x = Year, y = ratings_count)) +
geom_line(color = "purple", size = 1) +
labs(x = "Year", y = "Average ratings count", title = "Average ratings count of each book for each year") +
geom_smooth(method = "lm")
The total number of ratings each year seems to follow the pattern of the number of books published each year. However, the average ratings count for each book each year does not. We can see two greatest peaks of around 60,000 ratings in 1952 and 2019, along with various other smaller peaks. The average does not seem to follow any pattern.
sum_text_reviews_count_year <- aggregate(text_reviews_count ~ Year, data = goodreads_year, FUN = sum)
sum_text_reviews_count_year
ggplot(data = sum_text_reviews_count_year, aes(x = Year, y = text_reviews_count)) +
geom_line(color = "#9ACD32", size = 1) +
labs(x = "Year", y = "Total ratings count", title = "Total text reviews count of all books for each year") +
geom_smooth(method = "lm")
average_text_reviews_count_year <- aggregate(text_reviews_count ~ Year, data = goodreads_year, FUN = mean)
average_text_reviews_count_year
ggplot(data = average_text_reviews_count_year, aes(x = Year, y = text_reviews_count)) +
geom_line(color = "#9ACD32", size = 1) +
labs(x = "Year", y = "Average text reviews count", title = "Average text reviews count of each book for each year") +
geom_smooth(method = "lm")
Once again, the total number of text reviews count each year seems to follow the pattern of the number of books written each year. The average reviews count for each book each year reaches it peak in 2019, also along with some other smaller peaks. This time, we can observe that there is an overall increasing trend with respect to time, which means that as time goes, people are leaving more and more text reviews.